feat: multi-address DNS resolution for contact points and connections (DRIVER-201) - #890

Problem

DRIVER-201: when a contact point or a cluster node is given as a hostname that maps to multiple IPs (e.g. a DNS round-robin / dynamic-DNS entry), the driver only ever tried the first address — at initial contact, at connection time, and on control-connection reconnect. If that first IP was unreachable the driver raised AllNodesFailedException even though the hostname also resolved to healthy IPs.

This PR fixes DRIVER-201 end-to-end: every such hostname is expanded to all its addresses and each is tried in turn, for every connection the driver opens.

Note: this is a single consolidated PR for DRIVER-201. The work was originally split into #889 (Part 1 — expanding contact-point hostnames in the load-balancing query plan) and #890 (Part 2). #889's approach was an interim one that resolved DNS on the admin event loop under a timeout; it is subsumed here and #889 is closed. Because everything lands together, that interim JVM-DNS path never ships on its own.

Design

Name resolution is a connection-layer concern. ChannelFactory.connect() is the single place that turns "the address this node is known by" into "the addresses to actually try":

1. EndPoint.resolve() yields one address and does no lookup — it stays safe to call from an event loop.
2. If that address is a name, ChannelFactory expands it through the bootstrap's Netty AddressResolverGroup.
3. The candidates are tried in sequence until one connects.
4. The endpoint is pinned to the address that won, and the pinned copy is what the channel carries.

There is no public API change. EndPoint.resolve() keeps its signature and is not deprecated; third-party implementations keep working unchanged. Its javadoc gains one expectation: return the address as-is rather than looking names up, since resolution now happens in the connection layer.

There is, however, one behaviour change for callers of node.getEndPoint().resolve(), documented on resolve() and in the upgrade guide: for Cloud/SNI and client-route nodes the address is now the configured hostname, unresolved, so getAddress() returns null where it previously returned an IP. Nodes from system.peers and the control node are resolved as before. getHostString() covers both.

Expansion goes through Netty's resolver

Not through InetAddress.getAllByName(). That is the resolver an unresolved address already reached when it was handed to Bootstrap.connect(), so a custom AddressResolverGroup installed via NettyOptions.afterBootstrapInitialized() keeps applying, and Bootstrap.disableResolver() is still honoured. Whether an address needs resolving at all is the resolver's decision (isSupported() / isResolved()), exactly as in Bootstrap#doResolveAndConnect0 — a custom resolver may report an address that already carries an IP as unresolved in order to redirect it, and it still gets that say.

Consequence, unchanged from before this PR: with Netty's default resolver the lookup blocks the I/O event loop it runs on, because DefaultNameResolver calls InetAddress.getAllByName() inline. It is an I/O loop, never the admin loop that connect() is called from. Deployments that need non-blocking resolution can install DnsAddressResolverGroup and have it take effect — for the first time, for the SNI and client-route paths.

One Bootstrap and one EventLoop are picked per logical connect(), and each attempt takes a clone(eventLoop) of the bootstrap. Sharing one loop between resolution and the channel keeps the group's round-robin chooser advancing exactly once per connect (taking a loop for each would park every channel on half the loops), and it means the afterBootstrapInitialized() hook runs once per logical connection rather than once per address.

The candidate loop

• Each address is tried in turn; when all fail, the last error is propagated with every earlier failure attached as a suppressed exception, so no per-address cause is lost.
• Worst case is N × connect-timeout for a node with N addresses. Deliberate: failing on the first unreachable IP is what this ticket is about. Real DNS entries have few records.
• One failure is terminal rather than per-address: an UnsupportedProtocolVersionException against a node whose host id is known. Every address of an identified node is that same node, so replaying the whole negotiation ladder against each remaining IP buys nothing. An unidentified endpoint — a contact point, before host ids have been read — keeps going, since one name may expand to addresses of different nodes. That preserves what collapsing a name into a single Node would otherwise have removed: with advanced.resolve-contact-points = true each resolved address used to be its own Node, and ControlConnection advances its query plan on exactly this error.
• The starting address is rotated per name, so successive connections do not all pile onto the resolver's first record. Counters live on the ChannelFactory (i.e. per session) behind a 256-entry evicting cache: the names that reach it — contact points, the SNI proxy name, client-route hostnames — are not bounded by the configuration, since client routes can hand out different hostnames on every refresh.
• The queried name is re-attached to each candidate. A resolver may return results built from raw bytes, or labelled with a canonical/CNAME name of its own; that label would reach the pinned endpoint and hence be what DefaultSslEngineFactory / SniSslEngineFactory make TLS hostname verification check the certificate against. A nameless address is worse still — reading its host name triggers a blocking reverse lookup on the event loop and validation falls back to the IP or the PTR record. So the configured name always wins, which is also what happened before multi-address support, when Netty resolved only the TCP destination and the channel kept the original endpoint. Scoped IPv6 candidates keep their zone.

Pinning: PinnableEndPoint

A name describes a set of addresses, but a channel is connected to exactly one. ChannelFactory pins the endpoint to the address it used and hands that copy to the channel. This matters twice:

• Node identity. Once the driver has learnt over a connection that host id X answers at a given IP, that node must keep reconnecting to that IP; a node holding the multi-address endpoint could land on a different node later while still being treated as X. Pinning an identified node therefore deliberately removes its multi-IP fallback.
• No re-resolution on the channel path. SSL engine creation, GSSAPI service-name lookup and DefaultTopologyMonitor#savePort all call resolve() on the channel's endpoint. On a pinned copy that is a field read: it neither blocks on DNS nor risks a different address than the one the channel is on.

A pinned copy is otherwise indistinguishable from the original — same equals, hashCode, asMetricPrefix() and toString() — because nodes adopt pinned copies, and TaggingMetricIdGenerator tags node metrics with the endpoint's toString(). The pinned address is observable only through resolve(); which address a channel is on is in the channel's own toString(), which Netty builds from its remote address. PinnableEndPoint is internal: endpoints that do not implement it are left untouched.

Where the candidate list ends

ChannelFactory walks the candidates only while it is opening the channel. The control node's identity is read afterwards — a system.local query over the channel that won — so by the time that read can fail, the remaining addresses are gone.

Advancing the query plan there would write off a whole hostname on the strength of one of its addresses. With a single contact point and the default advanced.reconnect-on-init = false that meant initialization failed outright, and deterministically: a rebuilt session gets a fresh ChannelFactory whose rotation counters start at zero, so it lands on the same address every time while a healthy one sits unused.

ControlConnection therefore retries the same query plan entry instead, and the next attempt lands elsewhere because rotation advances once per connect. It arms only for a node with no host id whose endpoint denotes a name and whose channel reports a resolved pinned address — the same "addresses of an unidentified endpoint may belong to different nodes" reasoning as the terminal-failure rule above; an identified node stays pinned on purpose. The walk stops as soon as an address comes back round. MAX_ADDRESSES_PER_QUERY_PLAN_ENTRY (8) is only a backstop against a resolver that never repeats itself, and since it is tested before the address is recorded, one entry is attempted at most nine times.

The landed address is captured as soon as the channel opens, because resolveChannelNodeIfNeeded() overwrites the channel's endpoint with the one built from the system.local row before the registration step that can still fail.

Changes

Contact points stay unresolved

SessionBuilder / ContactPoints no longer resolve contact-point hostnames up front, so the query plan holds one unresolved node per contact point instead of one node per IP. advanced.resolve-contact-points is deprecated and has no effect. A hostname passed programmatically is expanded too, as long as the InetSocketAddress is unresolved (createUnresolved) — an already-resolved one is used as provided, which is what programmatic contact points did before this PR as well.

Endpoints

• DefaultEndPoint returns its address as-is, resolved or not, and implements PinnableEndPoint.
• SniEndPoint no longer re-resolves the proxy hostname on every resolve() call; the connection layer expands it, so all proxy A-records are tried within one attempt and a custom Netty resolver applies. A proxy hostname is stored unresolved whichever form it arrived in, so withCloudProxyAddress(new InetSocketAddress("proxy", 9042)) — which resolves eagerly — is not frozen on one proxy IP.
• ClientRoutesEndPoint returns the route's hostname unresolved instead of resolving it itself, so client-route hostnames are expanded by the connection layer too.

Control-connection reconnection query plan (folded from #889 review)

LoadBalancingPolicyWrapper.newControlReconnectionQueryPlan() composes the contact-point fallback as CompositeQueryPlan(regularPlan, new SimpleQueryPlan(contactNodes)) instead of mutating the policy's plan: built-in QueryPlans reject add()/addAll(), so the previous addAll(...) threw UnsupportedOperationException on every post-init control reconnect once the fallback defaulted on. The fallback is also kept when the live-node plan is empty, even for re-resolving topology monitors, so reconnection can still recover when there is nothing else to try.

NettyOptions.afterBootstrapInitialized

Contract documented: the driver installs its own handler afterwards, so a handler set by the hook is replaced (now warned about once), and the resolver configured there is what the driver expands names with.

OptionalLocalDcHelper

Removes the dead checkLocalDatacenterCompatibility() check. It warned when a contact point's datacenter differed from the configured local DC, but contact-point nodes never get a datacenter assigned during refresh, so it compared against null and could never reflect a real mismatch — while it could fire spuriously. The separate "configured local DC matches no node" warning is retained. Unrelated to the DNS fix itself; called out because it touches a protected extension point.

Tests

• ChannelFactoryNettyResolverTest — expansion through a custom resolver, disableResolver(), an already-resolved address passed through, a resolver redirecting an already-resolved address, resolution and connect sharing one event loop, a resolver throwing synchronously.
• ChannelFactoryMultiAddressTest — fallback across candidates with suppressed causes, per-name/per-session/bounded rotation, hostname re-attachment (nameless, CNAME-labelled, IPv6, scoped IPv6, no-name original), and the guards that fail the connect future instead of hanging it.
• ChannelFactoryPinnedEndPointTest, ChannelFactoryBootstrapHookTest, ChannelFactoryProtocolNegotiationTest — pinning, the hook contract, and the terminal-vs-retryable version rejection.
• DefaultEndPointTest / SniEndPointTest / ClientRoutesEndPointTest — resolve/pin semantics and pin-invisible identity; DefaultNodeTest — endpoint adoption and metric-updater rebuilds; AddressUtilsTest — name vs IP literal.
• LoadBalancingPolicyWrapperTest — real QueryPlan stubs (the earlier mutable LinkedList stub masked the crash), plus empty-plan and re-resolving-monitor cases.
• ControlConnectionTest — the same-node address walk: retry on identity-read failure and on a channel closing mid-resolve, termination when an address comes back round, the MAX_ADDRESSES_PER_QUERY_PLAN_ENTRY backstop, every failed address reported under the one node, and the two cases that must not retry (an IP literal, an already-identified node). The backstop case was checked to fail with the cap removed.
• MockResolverIT — end-to-end against a live cluster with a JVM-level DNS hook, including a multi-record name whose first-tried record is dead. That case captures ChannelFactory at DEBUG and requires the "trying next address" event for the dead record, so it fails rather than passing vacuously if the candidate ordering ever stops putting the dead record first.

Verified on JDK 11: full core unit suite (3862 tests), core + integration-tests install, javadoc:javadoc, a warning-free local docs build, a per-commit -Werror compile of all 9 commits, and MockResolverIT (3 tests) against live ScyllaDB.

History

Regrouped on 2026-08-06 into 9 per-concern commits, rebased onto the current scylla-4.x tip.

The branch had grown to 39 commits carrying roughly 1700 added lines that a later commit deleted again — the withdrawn EndPoint.resolveAll() API, a driver-owned resolver thread pool, and two test classes added and then removed. The repo rebase-merges, so all of that would have landed on scylla-4.x verbatim. Churn now equals the net diff exactly, no commit adds anything a later one deletes, and each commit compiles standalone under -Werror. The tree is unchanged by the regroup: git diff against the pre-regroup branch is empty.

The round-by-round detail lives in the review threads below. The larger course corrections, for the record:

• The first design added an EndPoint.resolveAll() API and had endpoints do their own JVM DNS. Withdrawn: it bypassed a custom Netty resolver, and it put a blocking lookup behind a public method the driver calls from an event loop. Resolution moved into ChannelFactory and the API addition was dropped, along with the resolve() deprecation and every @SuppressWarnings("deprecation") it had required.
• The interim query-plan-time expansion from fix: expand contact point hostnames to all DNS IPs at connection time (DRIVER-201) — Part 1/2 #889 (MetadataManager.getResolvedContactPoints(), its resolver executor and 3s timeout) is removed: connection-time expansion covers control-connection init and pool connections alike.
• A driver-owned resolver thread pool existed briefly and is gone with it — resolution runs on the channel's own event loop, so there is no pool to size or shut down.

On the suggestion that fallback-to-original-contact-points be reverted to false: its stated reason was that the flip "enables the blocking DNS fallback path by default", and that path no longer exists — the fallback appends unresolved hostnames and does no DNS at plan time. Moving resolution to the connection layer also makes the flip more necessary, not less: PinnableEndPoint binds the control node to the single address its connection reached, so it no longer re-expands, leaving the contact-point fallback as the only path back to a changed DNS record. TopologyMonitor.reresolvesNodeAddresses() documents that dependency.

Deliberately not fixed here:

• Duplicate resolver candidates are not collapsed, so a name with a repeated A record spends an extra connect-timeout on an address it already tried. Filed as Duplicate DNS records cause repeated connection attempts to the same address #989.
• The contact points appended by the reconnection fallback are not deduplicated against the live-node plan either, because at plan time they are hostnames while the live nodes are already-resolved IPs. Same shape as Duplicate DNS records cause repeated connection attempts to the same address #989; the cost is documented in reference.conf and the upgrade guide.
• DefaultNode.setEndPoint()'s clear→swap→build sequence is not atomic against concurrent metric writes: a write landing in the window goes through the cleared updater, which re-registers on demand, resurrecting one series. Strictly better than the previous ordering, which permanently deleted the new series; closing it properly means having clearMetrics() take the ids to clear rather than recomputing them, in every metrics implementation. Documented in the code.